1. Field of the Invention
The present invention relates to an image processing apparatus and image processing method which correct a blur of a capturing image caused by an imaging optical system.
2. Description of the Related Art
In an image capturing apparatus such as a digital still camera or digital video camera, an imaging optical system formed from a lens and the like guides light from an object onto a CCD or CMOS sensor serving as an image capturing device so as to form an image. The image capturing device converts the received light into an electrical signal. The electrical signal undergoes processing such as analog-to-digital (A/D) conversion and demosaicing necessary for converting an electrical signal into image data, thereby obtaining a capturing image.
Since the light arriving at the image capturing device has passed through the imaging optical system, the image capturing device influences the image quality of the capturing image. For example, when a high-power lens is used, a capturing image having a high resolution to its peripheral region can be obtained. On the other hand, if a low-cost low-power lens is used, the resolution of the capturing image conspicuously degrades especially in the peripheral region.
For example, when taking a picture of a starry sky, an image captured by a high-power lens shows each star almost as a point image. In an image captured by a low-power lens, however, each star does not appear as a point image but blurs. When taking a picture of a person, the use of a high-power lens enables one to obtain an image in which each hair is distinct. However, a low-power lens only obtains an image with a blurry mass of hair. That is, an image without definition is obtained by the low-power lens.
Such a blur depends on the characteristics of the imaging optical system and occurs even in an in-focus state. In other words, the resolution of the capturing image changes depending on the performance of the lens even in an in-focus state.
There is a method of correcting an image blur caused by the imaging optical system by performing image processing for the capturing image. In this method, image processing is executed based on the blur characteristic of the imaging optical system, which has been acquired as data in advance, thereby correcting an image blur caused by the imaging optical system.
To obtain the blur characteristic of the imaging optical system as data, for example, a PSF (Point Spread Function) is used. The PSF represents how a point of an object blurs. For example, a luminous body (point source) having a very small volume is captured via an imaging optical system in darkness. If an ideal imaging optical system is used, the light forms a point image on the surface (light-receiving surface) of the image capturing device. However, if an imaging optical system that yields a large blur is used, the light forms not a point image but an image spreading to some extent on the light-receiving surface. That is, the two-dimensional distribution of the light on the light-receiving surface corresponds to the PSF of the imaging optical system. When acquiring the PSF of the imaging optical system in actuality, it is not always necessary to capture an object such as a point source. Based on an image obtained by capturing, for example, a chart with white and black edges, the PSF can be calculated using a calculation method corresponding to the chart.
A method of using an inverse filter is known as a method of correcting an image blur by the PSF. Assume a case in which a point source is captured in darkness for the descriptive convenience. When an imaging optical system with a blur is used, light emitted by the point source forms a light distribution spreading to some extent on the light-receiving surface. The image capturing device converts the light into an electrical signal. The electrical signal is converted into image data, thereby obtaining a digital image of the point source. In the image obtained using the imaging optical system with a blur, not only one pixel corresponding to the point source has a significant, non-zero pixel value. Pixels in the neighborhood also have significant pixel values close to zero. Image processing of converting this image into an image having significant pixel values almost at one point is inverse filtering. The inverse filter allows to obtain an image as if it were captured using an imaging optical system with little blur.
A point source has been exemplified above for the sake of explanation. Regarding light from an object as an aggregate of a number of point sources, the blur of light coming from or reflected by each point of the object is eliminated, thereby obtaining an image with little blur even for an object other than a point source.
A detailed method of forming the inverse filter will be described next. A capturing image obtained using an ideal blur-free imaging optical system is defined as f(x,y), where x and y are two-dimensional pixel positions on the image, and f(x,y) is the pixel value of the pixel (x,y). A capturing image obtained using an imaging optical system with a blur is defined as g(x,y). The PSF of the imaging optical system with a blur is represented by h(x,y). Then, f, g, and h have a relation given byg(x,y)=h(x,y)*f(x,y)  (1)where * represents a convolution operation.
Correcting an image blur (to be referred to as blur correction hereinafter) amounts to estimating f obtained by the blur-free imaging optical system based on the image g obtained using the imaging optical system with a blur and h that is the PSF of the imaging optical system. When Fourier transformation is applied to equation (1) to obtain an expression in the spatial frequency domain, it can be written as a product for each frequency as given byG(u,v)=H(u,v)·F(u,v)  (2)where
H is an OTF (Optical Transfer Function) that is the Fourier transform of the PSF,
u is the spatial frequency in the x direction,
v is the spatial frequency in the y direction,
G is the Fourier transform of g, and
F is the Fourier transform of f.
To obtain the blur-free capturing image f from the blurred capturing image g, both sides of equation (2) are divided by H as perG(u,v)/H(u,v)=F(u,v)  (3)
F(u,v) obtained by equation (3) is inversely Fourier-transformed to the real space to obtain the blur-free image f(x,y). Let R be the inverse Fourier transform of 1/H. Convolution is performed in the real space to obtain a blur-free image byg(x,y)*R(x,y)=f(x,y)  (4)
R(x,y) of equation (4) is called an inverse filter. Actually, since a division by a divisor 0 occurs at the frequency (u,v) at which H(u,v) is zero, the inverse filter R(x,y) requires a slight modification.
Normally, the higher the frequency is, the smaller the value of the OTF is. The higher the frequency is, the larger the value of the inverse filter that is the reciprocal of the OTF is. Hence, convolution of the capturing image using the inverse filter enhances the high-frequency components of the capturing image, that is, noise (noise is a high-frequency component in general) contained in the capturing image. A method is known to impart a characteristic which does not enhance high-frequency components as much as the inverse filter by modifying R(x,y). A Wiener filter is famous as a filter that does not enhance high-frequency components so much in consideration of noise reduction.
As described above, it is impossible to completely remove a blur because of noise contained in the capturing image or a departure from an ideal condition such as existence of a frequency at which the OTF is zero. However, the above-described processing can reduce the blur. Note that filters such as an inverse filter and a Wiener filter to be used for blur correction will be referred to together as a “recovery filter” hereinafter. As a characteristic feature of the recovery filter, it performs image processing using the PSF of the imaging optical system, as described above.
To construct a Wiener filter, the OTF and noise amount of the imaging optical system are used. That is, as a characteristic feature, the Wiener filter is formed using the PSF or OTF and noise amount of the imaging optical system.
Note that the inverse filter and Wiener filter have been mentioned as the method of correcting a blur of the imaging optical system. Blur correction methods such as the maximum entropy method and Richardson-Lucy method have also been proposed in addition to those filters. They are equal to the Wiener filter in the sense that processing is performed based on the OTF or PSF and the noise amount to correct a blur of a capturing image and obtain a corrected image, although a detailed description thereof will be omitted.
Generally, frequency components contained in a capturing image are enhanced in blur correction. Since a capturing image contains noise, blur correction enhances the frequency components of the noise as well. For this reason, if the degree of blur correction is strong, noise also increases. To avoid the increase of noise, the degree of blur correction needs to be weakened. That is, the degree of blur correction and noise increase have a tradeoff relationship.
The Wiener filter is one of filters that implement optimum balance between the degree of blur correction and noise increase. More specifically, correction using a Wiener filter gives a corrected image closest in terms of square error to a capturing image containing neither noise nor a blur by the imaging optical system. That is, the Wiener filter performs correction in consideration of both noise increase and the degree of blur correction and can therefore maintain appropriate balance between noise increase and the degree of blur correction.
As described above, the Wiener filter can be calculated from the OTF and noise amount. Conventionally, the Wiener filter is constructed based on a noise amount measured under a predetermined capturing condition or an empirically defined noise amount. However, the noise amount varies in accordance with the capturing condition, and therefore, the noise amount measured under a predetermined capturing condition or the empirically defined noise amount is not accurate. As a result, the conventional Wiener filter does not necessarily have an optimum structure.
As an example of a noise amount variation according to the capturing condition, the temperature of the image capturing device rises so as to increase noise upon continuous shooting. A Wiener filter formed based on a noise amount smaller than the actual noise amount increases the noise of a capturing image more than necessary. Conversely, a Wiener filter formed based on a noise amount larger than the actual noise amount weakens the degree of blur correction more than necessary. In other words, blur correction based on an incorrect noise amount leads to insufficient correction or excessive noise.
In the above-described blur correction methods such as the maximum entropy method and Richardson-Lucy method as well, the degree of blur correction and noise increase have a tradeoff relationship. Hence, if no accurate noise amount is obtained, optimum blur correction cannot be done.